Well...Shit (Not car related)

"Lekker" (z34nismo) 08/06/2014 at 12:57 • Filed to: None

0

35

I work with a lot of technologies, and one of the ones I use the most is UNIX boxes for hospital's systems. Well I was given a task to basically delete a domain one of my clients wasn't using, so I went ahead and started. Well.... not sure how many of you do unix administration, but here's what happened: I was deleting some of the folders, and since it is multiple of them I just made a series of statements to do it for me, and then paste them onto the terminal. I do this a lot to document my commands, or to edit them easier. Well, I had done some manual deleting already, and moved to a folder where I had just deleted some info that ALSO contains vital stuff for another domain currently running on that node. Well, I ran my command, and apparently there was a TAB at the end of the command, which in Linux, auto-completes the path for the only available folder. Which in this case... was the domain they care about.

So after an hour of damn nearly crapping myself, I got a hold of the backup team and hopefully we will restore the filesystem I just hosed. Great. Now I feel like an idiot.

Have a dream machine for your trouble.

Nibbles > Lekker 08/06/2014 at 13:03

3

Thank gof for backups!

ttyymmnn > Lekker 08/06/2014 at 13:05

0

That's the problem with computers: They only do what you tell them to do.

Jake, Yes, wearing khakis > Lekker 08/06/2014 at 13:05

0

I know how you feel man.

I work on a majority of windows servers but ive done similar things.
Like upgrading raid card and backplane firmware while the raid set is degraded...
Needless to say that didnt go well, spent 70 hours of my own time restoring and fixing that

mcseanerson > Lekker 08/06/2014 at 13:07

0

First rule of tech club: Make backups
Second rule of tech club: Make backups

Lekker > ttyymmnn 08/06/2014 at 13:07

0

Well that is the thing, I didn't tell it to do anything. I simply left a tab at the end of a command, which translated to self-completion in unix. :( But alas, if I had done this right, one at a time, it wouldn't have happened

crowmolly > Lekker 08/06/2014 at 13:08

1

I worked with a guy who did rm -f * .tmp instead of rm -f *.tmp in an older version of unix once. In the root directory.

Oddly enough, this was the face he made after realizing what had just happened:

Lekker > Jake, Yes, wearing khakis 08/06/2014 at 13:09

0

Awe yikes, that sounds like a huge hassle. We actually have had some issues with our NIC's failing and not routing traffic to the other card, causing huge outages. Why are computers so complicated..oh wait.. bc we need a job.

BrainForest > Lekker 08/06/2014 at 13:10

0

Ohhh, what a nightmare. All you can do is just sit there and hope/pray/beg that there is a decent backup, and throw in a wish on the side that the backup guy has a sense of humor.

KusabiSensei - Captain of the Toronto Maple Leafs > Lekker 08/06/2014 at 13:11

0

At least you didn't do this:

# rm -rf / folder/to/delete

Jake, Yes, wearing khakis > Lekker 08/06/2014 at 13:11

1

Exactly, i love my job but i hate computers sometimes. But without computers id have no job and in turn no project cars. So i thank overly complicated computers for making me financially able to buy a old muscle car with no computers....

Lekker > crowmolly 08/06/2014 at 13:12

0

Oh dear god... We have horror stories of that happening in our company. But the greatest one I've heard: One dude deleted the ENTIRE ASP network, affecting 12,000 hospitals for 17hours, that was a 1.2$ billion outage, bc he thought it would be the local ASP tree for a client. Funnily enough my statement was: rm -fR /home/domainA, jsut because I don't like AT ALL going into any folder and running rm -fr *, it scares the bits out of me.

Lekker > BrainForest 08/06/2014 at 13:13

0

Thankfully the backup was done yesterday at 5pm, so there'll be little to no data loss. And there are literally so few people in this company with a sense of humor... I was just happy they backed that filesystem up. Lol

iwrock > Lekker 08/06/2014 at 13:14

0

That's why I got out of IT...

And remember kids, when you test your code, test it in production!

Lekker > KusabiSensei - Captain of the Toronto Maple Leafs 08/06/2014 at 13:14

0

Someone in my company did not too long ago, along with a chmod / 444.

Lekker > Jake, Yes, wearing khakis 08/06/2014 at 13:15

0

So much this

crowmolly > Lekker 08/06/2014 at 13:16

0

LOL. That is awful.

When I was learning UNIX a friend of mine in the class was writing a recursive test to create and delete processes via forking. You can see where this is going.

Well, of all the UNIX servers you could log in to he chose to log in to the university mail server because it was the fastest. Guess what happens when you create a runaway fork script on a mail server?

4 hours of outage and a 45 minute session on programming responsibility for the whole class, that's what.

Lekker > iwrock 08/06/2014 at 13:16

0

lmfao, this is at my coworker's desk:

BrainForest > iwrock 08/06/2014 at 13:18

0

That is very funny

ttyymmnn > Lekker 08/06/2014 at 13:18

0

I'm not a UNIX guy, or much of a computer guy for that matter, but it sounds like it did exactly what you told it to do when you left the tab in. I hope it all works out.

I used to work as a proctor in the Music Department computer lab, and we had a saying: There are two kinds of computer users: those who have lost data, and those who will . I once stupidly deleted an entire paper that I had been working on for hours. That sinking feeling.....

Lekker > crowmolly 08/06/2014 at 13:19

0

That sounds like a good time..........if you like that sort of thing. Yeah this is my first big boo-hoo at this job, so although there is no lectures to yell at me, I feel the disappointed scowl in the back of my head

Lekker > ttyymmnn 08/06/2014 at 13:23

0

Yeah it isn't a matter of if, rather a matter of when. Papers are the worst... you don't normally think to back up a papers.

JGrabowMSt > Lekker 08/06/2014 at 13:23

0

Because I have a nack for not exactly doing what either my boss or my clients want me to do (but they don't actually know that), in this situation, regardless of what I was asked, I would have basically removed the domain from the network without removing the data.

More often than not, I have customers who ask for work to be done, and then a week later bitch about how they can't get to the data.

Makes me look like a complete genius when it takes me 5 minutes to open a directory instead of 15 hours to run data recovery.

That and I also run a local backup before doing any work like that. eSATA or USB3 SSD means the image would be back on the server in a matter of minutes, even if there was a big borkup in the process somewhere. The 10-15 minutes of downtime I cause by running that backup is absolutely nothing in comparison, and often times if someone tries to tell me they don't need a backup, I tell them about other clients who didn't have a backup to begin with, and had controllers fail, or all sorts of wild things happen.

If it can go wrong, it will, and likely while I'm there trying to fix something unrelated.

Lekker > JGrabowMSt 08/06/2014 at 13:28

0

I totally agree with you on a normal circumstance. This domain hadn't been used in 2 years and had no clinical data, so it was easy to scrap. And thankfully the backup ran yesterday at 5pm which means no data will be lost. And yes, if something can go wrong, it will at the worst moment possible.

JGrabowMSt > Lekker 08/06/2014 at 13:31

0

Well, even for something like that, removing it from the domain, even if making the drive array larger, would be what I would push for. There will definitely be a day when some enterprising jackass is looking to get to some file he put on there.

ttyymmnn > Lekker 08/06/2014 at 13:31

1

That was early in my computer using, and in the days before multiple undo. I managed to hit Select All and delete the entire text, then I deleted another paragraph symbol because I'm anal about that sort of thing. When I realized what I had done, I hit Undo, and it only undid my last operation. The paper was gone. But when I did my doctoral dissertation, I had that sucker backed up on multiple disks and in multiple geographic locations. I was NOT going to lose that sucker.

Santiago of Escuderia Boricua > Lekker 08/06/2014 at 13:31

0

rm -fr /

Lekker > JGrabowMSt 08/06/2014 at 13:34

0

The good thing is that they don't have access to the back-end. Only the system engineers. But we do have backups of all data for up to 2 years, in case someone decides to be hard headed about something like that where it wouldn't be our responsibility. It wasn't removed to get space, it was removed bc our guidelines indicate after 1 year of no use a domain must be scrapped (it is contracted that way).

Lekker > ttyymmnn 08/06/2014 at 13:35

1

That is exactly what my dad did with his dissertation. Which was good bc the hard drive in his laptop went bad halfway through.

JGrabowMSt > Lekker 08/06/2014 at 13:41

0

Interesting. I don't specifically work in IT, so contracts like that are all greek to me. I do repairs for the most part, but once in a while I go places to test security. I 100% totally prefer just building/fixing machines more than the IT work. I make the toys that IT gets to play with, and I've made some very nice toys in the past couple years. I also really hate writing or checking scripts when they don't work right. It's just a bothersome process to me. Give me boxes and boxes of parts and a screwdriver, and I'll keep to myself.

Oh well. The more you know.

Lekker > JGrabowMSt 08/06/2014 at 13:56

1

Yeah, i am with you on that. Entirely. But you have to start somewhere in order to move to that position :) I love scripting. Debugging someone's script on the other hand, might as well shoot me.

KusabiSensei - Captain of the Toronto Maple Leafs > Lekker 08/07/2014 at 07:08

0

Since the -R flag on chmod isn't set, that should only affect the root of the file system. What you've set isn't all that unusual unless it's done recursively.

If it was done recursively, then that's a bit...yeah...overzealous IS.

thebigbossyboss > Lekker 08/07/2014 at 08:44

0

Once I hit the wrong button on the work DVR and deleted about a weeks worth of recordings on there. The sad thing was it takes like 5 minutes to delete, and you can't stop it so you're just looking at this thing deleting things you need and are going "god damn it" the whole time.

It's like watching a slow motion ship wreck.

Lekker > KusabiSensei - Captain of the Toronto Maple Leafs 08/07/2014 at 09:50

0

yeah we add custom parameters to our commands, -R is to recursively delete encrypted data. Which.... it did.. But thankfully after 9 hours I was able to restore the system

Lekker > thebigbossyboss 08/07/2014 at 09:51

0

Gahh that has GOT to be like watching your arm getting sawed off. What kind of material is it?

thebigbossyboss > Lekker 08/07/2014 at 10:34

0

Probably can;t say.

Let's just say...the material was gone. It's gone forever.